MIR Benchmarking: Lessons Learned from the Multimedia Community

Music Information Retrieval may be perceived as part of the larger Multimedia Information Retrieval research area. However, many researchers in Music Information Retrieval are unaware that the problems they deal with have analogous problems in image and video retrieval. Many issues concerning the creation of testbed digital libraries and effective benchmarking of information retrieval systems are common to all multimedia retrieval systems. We examine the approaches used in the image and video communities and show how they are applicable to testbed creation and information retrieval system evaluation when the media is music.